Big data is the term used to describe massive volumes of both structured and unstructured data that are so large that they are difficult to process using traditional database and data processing techniques. However, big data is becoming increasingly important, as the volume of data available grows rapidly due to the near ubiquitous availability of the Internet and data-generating devices, such as mobile phones and tablet computers. In addition, with the rapid growth of big data has come the recognition that the analysis of larger data sets can lead to more accurate analysis.
This is particularly true for transaction-related data, wherein a transaction is an exchange or interaction between two entities. Transaction data will therefore comprise a record of each transaction that defines the source and destination of the transaction, and further details of the transaction such as the date and/or time of the transaction, the content/subject of the transaction, the size/volume of the transaction etc. The processing and analysis of the transaction related data can therefore be extremely useful for identifying deviations or anomalies from normal or expected patterns of transactions that can be indicative of issues arising within the system that generated the transactions. For example, transactions for which such analysis would be particularly useful may include financial, accounting, insurance, security trading, security access and phone or computer mediated transactions.
A particular challenge when dealing with vast amounts of transaction-related data involves the visualization of this data, and the interrogation of such visualizations for analysis purposes. Generating visualizations of such vast amounts of data will typically require a significant amount of processing power, otherwise the time taken to generate a visualization will be too long to be of sufficient use. This is especially problematic when dynamic interrogation of a visualization is necessary in order for a worthwhile analysis to be performed. In addition, when visualizing vast amounts of data, clutter quickly becomes a problem due to the physical limitations (e.g. size and resolution) of the screens on which visualizations can be displayed, such that a large proportion of the detail of the individual transactions is lost. It can then become extremely difficult to extract any useful information from a data visualization. Furthermore, even if the size and/or resolution of the displays are increased, thereby providing more pixels with which to display a data visualization, there is a limit to human visual cognition, both in terms of the size of image that we can process and our visual acuity. Consequently, the generation of visualizations of vast amounts of transaction-related data is far from trivial, and increasing the efficiency with which a visualization is generated and the efficiency with which a visualization can convey information is highly desirable and technically challenging.